executor: Fix index join hash produces redundant rows for left outer anti semi join type #52908

yibin87 · 2024-04-26T07:19:46Z

What problem does this PR solve?

Issue Number: close #52902

Problem Summary:
In IndexHashJoin inner work, we use join type to check if redundant matched rows should be appended to JoinResult. However, only semi and leftOuterSemi is recongnized:

tidb/pkg/executor/index_lookup_hash_join.go

Lines 739 to 752 in 8ba1fa4

    
           isSemiJoin := joinType == plannercore.SemiJoin || joinType == plannercore.LeftOuterSemiJoin 
        
           for ; matchedOuterEntry != nil; matchedOuterEntry = matchedOuterEntry.next { 
        
           	ptr := matchedOuterEntry.ptr 
        
           	outerRow := task.outerResult.GetRow(ptr) 
        
           	ok, err := codec.EqualChunkRow(iw.ctx.GetSessionVars().StmtCtx.TypeCtx(), innerRow, iw.hashTypes, iw.hashCols, outerRow, iw.outerCtx.hashTypes, iw.outerCtx.hashCols) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	if !ok || (task.outerRowStatus[ptr.ChkIdx][ptr.RowIdx] == outerRowMatched && isSemiJoin) { 
        
           		continue 
        
           	} 
        
           	matchedRows = append(matchedRows, outerRow) 
        
           	matchedRowPtr = append(matchedRowPtr, chunk.RowPtr{ChkIdx: ptr.ChkIdx, RowIdx: ptr.RowIdx}) 
        
           }

While anti-semi join doesn't care matched rows, left outer anti semi will append joined rows, thus caused the problem.

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: yibin <[email protected]>

tiprow · 2024-04-26T07:20:04Z

Hi @yibin87. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yibin87 · 2024-04-26T07:20:26Z

/cc @windtalker @XuHuaiyu

codecov · 2024-04-26T07:37:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.5586%. Comparing base (5872ccd) to head (bc516e1).
Report is 105 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #52908         +/-   ##
=================================================
- Coverage   72.2059%   56.5586%   -15.6473%     
=================================================
  Files          1469       1600        +131     
  Lines        427109     618176     +191067     
=================================================
+ Hits         308398     349632      +41234     
- Misses        99536     244855     +145319     
- Partials      19175      23689       +4514

Flag	Coverage Δ
integration	`37.0721% <100.0000%> (?)`
unit	`71.2337% <100.0000%> (+0.1354%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`49.1682% <ø> (+8.0714%)`	⬆️

windtalker

LGTM

ti-chi-bot · 2024-04-26T09:44:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XuHuaiyu,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-04-26T09:45:01Z

[LGTM Timeline notifier]

Timeline:

2024-04-26 07:31:18.556528511 +0000 UTC m=+331835.296431424: ☑️ agreed by XuHuaiyu.
2024-04-26 09:45:00.520587508 +0000 UTC m=+4854.277723078: ☑️ agreed by windtalker.

ti-chi-bot · 2024-04-26T10:12:54Z

In response to a cherrypick label: new pull request created to branch release-8.1: #52923.

ti-chi-bot · 2024-04-26T10:12:57Z

In response to a cherrypick label: new pull request created to branch release-7.5: #52924.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-26T10:13:00Z

In response to a cherrypick label: new pull request created to branch release-7.1: #52925.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-26T10:13:04Z

In response to a cherrypick label: new pull request created to branch release-6.5: #52926.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-26T10:13:07Z

In response to a cherrypick label: new pull request created to branch release-6.1: #52927.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-26T10:13:12Z

In response to a cherrypick label: new pull request created to branch release-5.4: #52928.

…anti semi join type (#52908) (#52923) close #52902

…anti semi join type (#52908) (#52924) close #52902

…anti semi join type (#52908) (#52925) close #52902

…anti semi join type (#52908) (#52927) close #52902

…anti semi join type (#52908) (#52926) close #52902

…anti semi join type (#52908) (#52928) close #52902

Fix issue 52902 by recongnizing anti semi and anti semi left outer join

bc516e1

Signed-off-by: yibin <[email protected]>

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 26, 2024

ti-chi-bot bot requested review from windtalker and XuHuaiyu April 26, 2024 07:20

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Apr 26, 2024

XuHuaiyu approved these changes Apr 26, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 26, 2024

windtalker approved these changes Apr 26, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 26, 2024

ti-chi-bot bot merged commit 944fff5 into pingcap:master Apr 26, 2024
21 of 23 checks passed

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52923

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 26, 2024

This is an automated cherry-pick of pingcap#52908

2a0daff

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52924

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 26, 2024

This is an automated cherry-pick of pingcap#52908

6bd11f6

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52925

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 26, 2024

This is an automated cherry-pick of pingcap#52908

7146589

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52926

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 26, 2024

This is an automated cherry-pick of pingcap#52908

b21c295

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52927

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 26, 2024

This is an automated cherry-pick of pingcap#52908

dc4451a

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 26, 2024

executor: Fix index join hash produces redundant rows for left outer anti semi join type (#52908) #52928

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

a8bb5be

…anti semi join type (#52908) (#52923) close #52902

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

0359bbc

…anti semi join type (#52908) (#52924) close #52902

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

8b15640

…anti semi join type (#52908) (#52925) close #52902

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

5716835

…anti semi join type (#52908) (#52927) close #52902

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

ddc6543

…anti semi join type (#52908) (#52926) close #52902

ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024

executor: Fix index join hash produces redundant rows for left outer …

01258a9

…anti semi join type (#52908) (#52928) close #52902

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: Fix index join hash produces redundant rows for left outer anti semi join type #52908

executor: Fix index join hash produces redundant rows for left outer anti semi join type #52908

yibin87 commented Apr 26, 2024 •

edited

Loading

tiprow bot commented Apr 26, 2024

yibin87 commented Apr 26, 2024

codecov bot commented Apr 26, 2024 •

edited

Loading

windtalker left a comment

ti-chi-bot bot commented Apr 26, 2024

ti-chi-bot bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

	isSemiJoin := joinType == plannercore.SemiJoin \|\| joinType == plannercore.LeftOuterSemiJoin
	for ; matchedOuterEntry != nil; matchedOuterEntry = matchedOuterEntry.next {
	ptr := matchedOuterEntry.ptr
	outerRow := task.outerResult.GetRow(ptr)
	ok, err := codec.EqualChunkRow(iw.ctx.GetSessionVars().StmtCtx.TypeCtx(), innerRow, iw.hashTypes, iw.hashCols, outerRow, iw.outerCtx.hashTypes, iw.outerCtx.hashCols)
	if err != nil {
	return nil, nil, err
	}
	if !ok \|\| (task.outerRowStatus[ptr.ChkIdx][ptr.RowIdx] == outerRowMatched && isSemiJoin) {
	continue
	}
	matchedRows = append(matchedRows, outerRow)
	matchedRowPtr = append(matchedRowPtr, chunk.RowPtr{ChkIdx: ptr.ChkIdx, RowIdx: ptr.RowIdx})
	}

executor: Fix index join hash produces redundant rows for left outer anti semi join type #52908

executor: Fix index join hash produces redundant rows for left outer anti semi join type #52908

Conversation

yibin87 commented Apr 26, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Apr 26, 2024

yibin87 commented Apr 26, 2024

codecov bot commented Apr 26, 2024 • edited Loading

Codecov Report

windtalker left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Apr 26, 2024

ti-chi-bot bot commented Apr 26, 2024

[LGTM Timeline notifier]

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

ti-chi-bot commented Apr 26, 2024

yibin87 commented Apr 26, 2024 •

edited

Loading

codecov bot commented Apr 26, 2024 •

edited

Loading